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1.  INTRODUCTION 

The  United  States  Navy  purchases  millions  of  dollars  worth  of  major 
caliber  ammunition  each  year  to  supply  its  warships.  These  ships  have 
missions  ranging  from  air  defense  to  gunfire  support  of  amphibious  forces  on 
land.  The  rounds  of  ammunition  purchased  are  each  individual  systems 
subject  to  various  forms  of  failure,  and  are  purchased  by  component.  There 
currently  exists  no  systematic  method  for  determining  the  minimum 
acceptable  reliability  level  for  round  components,  reliability  levels  are 
currently  established  rather  arbitrarily,  but  with  some  total  system  reliability 
in  mind. 

The  reliability  of  each  of  the  components  impacts  the  effectiveness  of  the 
weapon  system  and  subsequently  the  effectiveness  of  the  battle  force.  In  order 
to  procure  and  maintain  ammunition  which  will  provide  adequate  utility  to 
Naval  forces,  there  must  be  a  clear  understanding  of  this  relationship  of 
ammunition  reliability  and  force  effectiveness  in  various  missions. 

Consider  a  single  round  of  major  caliber  ammunition  which  is  comprised 
of  a  small  number  of  components,  including  fuses,  primers,  propellant 
igniters,  propellant,  propellant  casing,  case  plug,  projectile  and  explosive. 
Each  of  these  major  components  is  in  turn  an  assembly  of  a  number  of 
subcomponents.  A  round  may  experience  failures  in  each  of  these 
subcomponents.  While  some  of  the  possible  failures  result  in  preventing  an 
effective  round  from  arriving  at  the  target,  other  failures  prevent  the  gun  or 
gun  cluster  from  being  fired  for  some  period  after  the  failure.  In  relating 
reliability  performance  to  battle  force  performances,  attention  must  be  paid  to 


2/22/91.11:38  AM.l 


time  sensitive  situations  such  as  self-defense.  The  proper  measure  of 
effectiveness,  and  implied  optimal  reliability,  must  take  into  account  the  cost 
of  extra  reliability,  the  value  of  the  target,  and  the  impact  of  modes  of  failure 
on  performance. 

The  objective  of  this  research  is  to  establish  reliability  performance 
measures,  and  to  specify  minimum  reliability  goals  for  major  caliber 
ammunition  used  in  Naval  Gunfire  Support  (NGFS)  system.  The  reliability 
of  ammunition  and  gun  system  must  be  expressed  in  terms  of  battle  goals 
which  include  mission  time,  average  casualty  rate  over  mission  lifetime,  and 
the  percentage  of  opponent's  destructive  power  disabled  at  arbitrary  time  D. 
Then  the  decision  maker  has  the  ability  to  measure  reliability  performance  in 
terms  of  the  battle  goals  with  which  he  is  used  to  dealing. 

The  approach  we  took  in  this  research  is  described  in  Figure  1.1.  We  first 
developed  a  simple  measure  called  ef  which  is  a  function  of  reliability 
measures  of  individual  components  of  the  ammunition  and  gun  system. 
The  (theoretical)  ef  measure  represents  the  expected  number  of  effective 
rounds  per  unit  time.  The  round  is  considered  to  be  effective  if  it  gets  out  of 
the  barrel,  hits  the  target  and  detonates  successfully. 

In  order  to  determine  reasonable  reliability  thresholds  for  ammunition, 
we  must  analyze  the  integrated  system  involving  rounds,  guns,  targeting  and 
scheduling.  We  compiled  the  data  from  various  sources  including  Joint 
Munitions  Effectiveness  Manuals  (JMEMS)  and  the  OP-03  Material  Readiness 
Data  Base  (MRDB)  on  the  component  reliability  and  gun  reliability.  This 
information  is  used  to  generate  the  NGFS  scenario  data  file  for  simulation. 
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Figure  1.1.  Flow  Diagram  of  Reliability  Goal  Determination 

We  developed  a  simulation  model  to  find  the  relationship  between  ef 
values  and  the  battle  goals  which  we  call  standard  performance  measures. 
The  model  is  designed  to  simulate  the  NGFS  scenarios  to  mimic  the  spotting, 
communication  and  targeting.  The  simulation  results  show  that  the  ef 
values  are  highly  correlated  with  the  standard  performance  measures.  They 
explain  variability  in  observations  of  the  standard  performance  measures  for 
different  component  reliability  configurations.  Thus  the  decision  maker  is 
able  to  determine  the  effectiveness  of  ammunition  in  terms  of  battle  goals. 
He  can  find  the  minimum  (or  critical)  ef  value  which  is  still  mission 
effective. 
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This  will  support  his  decision  making  process  on  procurement  and 
surveillance.  In  procurement  decisions,  he  can  specify  reliability  values  for 
components  and  subcomponents  which  meet  minimum  requirements  for  ef 
in  a  most  economical  way.  Ammunition  surveillance  is  the  practice  of 
removing  several  rounds  from  a  stockpiled  lot  of  ammunition  and  testing 
the  reliability  of  the  rounds  or  of  the  components.  He  then  can  decide 
whether  or  not  to  perform  some  sort  of  rework  on  an  ammunition  lot  to 
improve  the  reliability  to  bring  it  to  mission  effective  status.  Currently  the 
acceptance  criteria  for  pronouncing  a  stockpiled  lot  fleet-ready  are  ad  hoc.  Our 
research  proposes  replacing  the  current  surveillance  test  criteria  with  those 
that  will  ensure  that  the  lot  tested  exceeds  minimum  ef  in  efficiency. 

The  results  of  our  research  can  also  be  utilized  to  study  the  impact  on 
marginal  incremental  improvements  in  component  reliability  for  contractors 
or  procurement  agents.  We  provide  an  important  set  of  tools  useful  in 
managing  procurement  and  surveillance  of  ammunition. 

The  structure  of  this  report  is  as  follows.  In  Section  2,  we  develop  a 
measure  of  performance  of  the  gun-round  system,  ef,  which  represents  the 
number  of  effective  rounds  per  unit  time.  A  simulation  model  is  described 
in  Section  3,  which  is  used  to  estimate  the  effectiveness  of  NGFS  given  the  ef 
values.  In  Section  4,  we  explain  how  to  run  the  simulation  program,  and  the 
statistical  analysis  of  simulation  results.  We  also  describe  in  some  detail  the 
variance  reduction  techniques  applied  to  our  simulation  study,  which  results 
in  substantial  improvement  of  simulation  output.  Section  5  includes  the 
conclusions  and  future  research. 
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2.  MEASURING  PERFORMANCE  OF  MAJOR  CALIBER  AMMUNITION 

In  this  section  we  begin  our  exploration  of  the  performance  characteristics 
of  the  naval  gunfire  support  system.  In  particular,  we  want  to  establish  an 
appropriate  performance  measure  to  predict  variations  in  the  effectiveness  of 
naval  gunfire  with  respect  to  changes  in  round  component  reliabilities. 

The  execution  of  an  amphibious  assault  by  combined  naval  and  marine 
forces  can  be  conceptually  divided  into  two  distinct  phases.  Phase  one,  which 
we  shall  call  preemptive  shelling,  occurs  prior  to  the  landing  of  any  marine 
forces.  The  objective  of  preemptive  shelling  is  to  reduce  the  capabilities  of 
enemy  shoreline  defenses  by  bombarding  defense  positions.  This 
bombardment  is  scheduled  and  directed  by  naval  forces.  The  schedule  is 
static;  it  does  not  change  once  the  phase  commences.  Successful  execution  in 
the  preemptive  shelling  phase  will  increase  the  probability  of  success  in  the 
second  phase,  which  we  call  assault  support.  The  transition  to  the  assault 
support  phase  occurs  when  marine  forces  land  and  take  their  initial  positions. 
The  assault  support  phase  is  characterized  by  the  continual  receipt  of  requests 
for  bombardment  of  specific  shore  defense  positions.  Calls  for  assault  support 
shelling  are  made  by  the  landed  force  and  supported  by  spotters  stationed  near 
the  targets.  These  calls  for  support  shelling  arrive  intermittently  and  are 
serviced  based  on  the  priority  of  the  call  and  the  destructive  power  of  the 
target. 
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2.1   MAJOR  CALIBER  AMMUNITION 

Consider  a  single  round  of  major  caliber  ammunition.  It  is  comprised  of 
a  small  number  of  major  components,  including  fuses,  primers,  propellant 
igniters,  propellant,  propellant  casing,  case  plug,  projectile,  and  explosive. 
Each  of  these  major  components  is  in  turn  an  assembly  of  a  number  of 
subcomponents. 
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Figure  2.1.  5-Inch,  54-Caliber  Propelling  Charge  and  Projectile 

The  key  functional  characteristic  of  this  system  is  its  strictly  serial 
operation.   For  example,  a  typical  detonating  train  consists  of  a  chain  of  stages 
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intended  to  increase  the  fuze  signal  energy  level  to  a  degree  sufficient  to 
detonate  the  explosive  payload.  The  output  signal  of  the  sensing  equipment 
(radar  signal,  acoustic  impulses,  hydrostatic  pressure,  etc.)  is  sent  to  a  primer. 
This  device  converts  electrical  or  mechanical  energy  into  explosive  energy. 
Although  this  energy  is  minimal,  it  suffices  to  trigger  an  auxiliary  detonator. 
Following  this,  the  output  power  of  the  auxiliary  detonator  triggers  a  chain  of 
insensitive  explosives  in  the  booster  stage  until  detonation  of  the  payload 
occurs.  (See  Weapons  Systems  Fundamentals,  NAVORD  OP  3000  Vol.  2,  First 
Revision,  for  details.)  Then  for  example,  if  the  primer  fails,  no  other 
component  down  the  chain  will  have  a  chance  to  operate.  It  is  this  simple 
serial  structure  that  greatly  simplifies  the  analysis  of  the  possible  failure 
modes  induced  by  the  system,  as  it  will  be  seen  in  the  following  subsection. 

2.2  FAILURES 

We  will  continue  now  with  a  structural  analysis  of  the  common  modes 
of  failure  of  the  single  gun  round  system.  We  will  consider  a  round  to  be 
effective  if  it  gets  out  of  the  barrel,  hits  the  target  and  detonates  successfully. 
Otherwise,  if  any  of  these  conditions  is  not  satisfied,  the  round  will  be  labelled 
as  non-effective.  There  are  two  operationally  distinct  failure  classes,  namely, 
interruptive  failures  and  non-interruptive  failures.  We  will  assume  that 
there  is  a  total  of  N  different  types  of  failures  with  type  F\  through  type  F/v-i 
corresponding  to  interruptive  failures  and  type  F^  corresponding  to  a  non- 
interruptive  failure. 

An  interruptive  failure  of  type  i  (Fz,  i  =  1,  ...,  N-l)  is  caused  by  a 
malfunction  of  the  round,  the  gun,  or  the  crew,  which  causes  the  gun  to  stop 
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firing  for  some  time  T,.  Examples  of  interruptive  failures  are  firing  pin 
failures,  or  failures  in  the  propellant  chain.  An  interruptive  failure  causes 
the  system  to  stop  firing  for  some  time  T,  in  order  to  fix  the  problem,  after 
which  the  system  performs  as  before. 

A  non-interruptive  failure  occurs  when  the  round  is  successfully  fired 
from  a  gun  barrel  but  fails  to  cause  effective  damage  to  the  target.  Causes  for 
this  might  be  a  faulty  or  incorrectly  set  variable  time  fuse,  insufficient  initial 
velocity,  guidance  errors,  faulty  detonation  or  even  targeting  errors  caused  by 
the  crew.  Although  no  effective  damage  is  done  to  the  target,  a  non- 
interruptive  failure  does  not  affect  the  gun  in  any  other  way,  i.e.,  a  non- 
interruptive  failure  does  not  affect  the  future  firing  rate  of  the  gun. 

For  the  current  analysis  we  have  assumed  that  T, ,  i  =  1,  ...,  N  are  known 
and  deterministic  values.  (We  plan  to  extend  this  work  to  include  randomly 
distributed  repair  times).  We  will  also  normalize  these  values  according  to 
the  firing  rate  of  the  gun,  i.e.,  if  the  gun  is  firing  at  18  rounds/minute,  we 
then  measure  T,  in  units  of  1/18  minutes.  In  this  manner  the  down  times 
caused  by  each  particular  failure  will  be  weighted  proportionally  to  the  firing 
rate  of  the  gun,  so  that  a  10-minute  interruption  is  considered  worse  for  a  gun 
shooting  at  a  high  firing  rate  than  a  gun  shooting  at  a  lower  rate.  This  also 
implies  that  an  effective  round  and  a  non-interruptive  failure  each  take  one 
unit  of  time. 

Consider  again  a  single  round  of  a  major  caliber  ammunition.  The  linear 
nature  of  this  system  allows  us  to  model  the  system  so  that  the  lower  levels  of 
failure  take  precedence  over  the  higher  levels.  Then,  the  failure  (conditional) 
probabilities  that  can  be  estimated  by  observation  are  given  by 
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x\  =  P[F\  occurs] 

Xi  =  P[Fi  occurs  |  no  Fj  occurred  for  j  <  i], 

for  i  =  2,  ...,  N.    Hence,  the  probabilities  of  the  different  failure  types  may  be 
computed  as 

(\i  -  P[Fi  occurs] 

qx  =  P[F{  occurs  and  no  Fj  occurs  for ;  <  i]. 

By  the  way  we  have  defined  the  observed  conditional  probabilities  (x/s)  the 
failure  types  are  independent  and  therefore 

qi=xiY[(l-xj)  (2.1) 


1 


for  i  =  1,  ....  N.    Thus,  the  qi  values  will  allow  us  to  compute  the  reliability 
measure  for  the  single  gun-round  system. 

2.3  A  MEASURE  OF  PERFORMANCE  OF  THE  GUN-ROUND  SYSTEM 

We  now  present  a  measure  of  performance  of  the  gun-round  system 

based  on  the  time  required  to  deliver  an  effective  round  on  the  target.    Recall 

that  a  round  is  considered  effective  upon  successful  detonation  on  the  target. 

Let  the  random  variable  Tp  represent  such  time  measured  in  firing  epochs, 

then 

N 
TP  =  YlPiTi}  +  1  (2-2) 
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where  P,  is  the  number  of  failures  of  type  i  occurring  between  effectively 
delivered  rounds,  i.e.,  the  Pj's  are  geometric  random  variables  with  parameter 
Pi  =  1  -  c\[,  for  i  =  1, ...,  N.  Thus,  the  first  two  moments  of  Tp  are  given  by 

N 


tp]=X^(i-Pi)/Px+i; 
1=1 

N 
vai[Tp]  =  ^(l-Pi)pf. 


(2.3) 
(2.4) 


There  is  a  very  important  relationship  between  these  two  moments  that 
serves  to  consolidate  our  objectives,  namely,  monotonicity.  As  it  will  be  seen 
in  the  following  theorem,  the  expected  value  and  the  variance  of  the  random 
variable  Tp  increase  and  decrease  together.  In  evaluating  the  performance  of 
a  given  set  of  components  with  associated  X{  values,  low  variance  of  Tp  is 
desirable  as  well  as  low  expected  value.  This  theorem  guarantees  that 
components  giving  lowest  expected  time  between  effective  rounds  also  give 
the  lowest  variance.  Thus,  we  are  interested  in  decreased  values  of  both  E[Tp] 
and  var[Tp],  and  these  goals  do  not  conflict  in  any  way. 


THEOREM  2.1.  Given  two  alternatives  p  and  p  in  [0,  1]  ,  if  E 


71 


<E 


T- 

[7 


,  then 


var 


T„ 


<  var 


T- 


PROOF.    We  will  show  that  for  two  points  p  and  p  suitably  close  to  one 
another,  that  VETp     (p-p)  and  Vvar  T„     [p~p)  have  the  same  sign.    Thus, 


descent  directions  for  E 
the  product 


r„ 


are  also  descent  directions  for  var 


T 
L  PJ 


Consider 
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T 

=  (p-p)  A(p-f) 


(2.5) 


which  is  a  quadratic  form  in  the  matrix  A.    It  is  simple  to  determine  that  A  is 
positive  semi-definite.    This  tells  us  that 


VE 


-iT  T 

[Tp\    (P-P)         Vvar(^)    (P-P) 


>0 


(2.6) 


for  any  vector  [p-p).     Hence,  we  may  conclude  that  in  a  suitably  small 


r„ 


are  also  descent 


neighborhood  of  any  point  p,  descent  directions  for  E 
directions  for  van  T    .    [] 

Backed  by  the  results  of  the  above  theorem  we  can  now  define  a  measure 
of  performance  of  the  gun-round  system  as: 

ef  =  1  /  E(Tp  J  =  round  efficiency.  (2.7) 

This  quantity  represents  the  number  of  effective  rounds  per  unit  time  (again 
measured  in  firing  epochs).     As  will  be  seen  in  the  following  section  the 
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simulation  results  confirm  that  ef  represents  a  good  measure  of  performance 
of   the   naval   gunfire   support   system,   highly   correlated   with   both   the 
destruction  time  of  the  target  and  the  destructive  power  left  by  time  t. 
Example  1 

Eight  failure  types  known  to  cause  a  preponderance  of  the  failures 
experienced  by  a  naval  gun  are  given  below  in  Table  2.1  along  with  their 
interruption  (repair)  times.  These  times  were  derived  from  studying  the 
figure  "Hot  Gun  Misfire  Procedures  and  Safety  Time  Schedule  for  5-inch,  54- 
caliber  Guns  Mark  45  Mods  0  and  1".  Failure  type  descriptions  are  given  in 
Table  2.2,  with  some  symptom  information,  remedy,  and  the  interruption 
time  for  each.  When  two  numbers  are  provided  they  are  for  hot  and  cold  gun 
situations.  Repair  of  these  types  of  failures  on  a  hot  gun  require  that  the  gun 
be  cooled  for  30  minutes  before  repair  can  begin.  This  assumes  that  an 
automatic  internal  cooling  system  can  be  employed.  The  wait  is  two  hours 
with  no  cooling  system.  Finally  we  evaluate  the  efficiency  of  a  round  under 
these  conditions. 

TABLE  2.1 


Failure  Type 

Description 

Data  Status 

1 

Loader  Microswitch  Fails 

E 

2 

Firing  Circuit  Fails 

E 

3 

Breachblock  Closed 

E 

4 

Firing  Pin  Fails 

E 

5 

Propellant  Primer  Fails 

D 

6 

Propellant  Impurity 

D 

7 

Fuse  Fails  or  is  Missed 

D 

8 

Detonation  Train  Fails 

D 
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The  qualifier  data  status  is  E  if  the  probability  of  failure  must  be  estimated 
from  data  provided  from  a  gun  failure  data  base,  while  those  with  data  status 
D  are  failure  types  whose  probabilities  are  variable,  open  to  decision. 

TABLE  2.2 


Fi 

Description/Symptom 

Remedy 

Ti 

Pi 

1 

Round  jammed,  loader  stops 

Mechanically 
unjam 

25/55 

0.05 

2,3 

Electronic  failure  in  system 

Isolate  and 

replace 

component 

1.5 

0.1 

4 

Electronics  work  but  no  fires 
occur 

e.g.,  Replace  pin 

21/51 

0.02 

5 

Propellant  will  not  fire  though 
mechanical  firing  sequence 
occurs 

Eject  propellant 
casing  and  fire 
clearing  round 

10 

0.03 

6,7,8 

Non-interruptive  failure 

Shoot  again 

1 /firing 
rate 

0.01 

Based  on  the  above  data  and  the  equations  provided  in  the  previous 
subsection  we  evaluated  the  efficiency  of  the  round  for  a  gun  shooting  at  20 
rounds/minute  and  obtained  ef  =  0.16,  i.e.,  16%  of  each  round  is  effective.  In 
other  words,  if  we  assume  that  10  rounds  are  necessary  to  destroy  the  target, 
then  given  ef  =  0.16,  we  will  need  an  average  of  62.5  firing  epochs  to  deliver 
10  effective  rounds. 

2.4  PURPOSE  OF  ef 

As  it  will  be  seen  in  the  following  section,  the  simulation  model  (and  the 
associated  computer  program)  will  allow  us  to  determine  the  appropriate 
value  of  ef  in  order  to  meet  battle  goals  with  respect  to  the  time  required  to 
destroy  all  targets,  the  casualty  rate,  destructive  power  left  by  time  D,  etc.  The 
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selected  number  for  ef,  say  ef*,  will  thus  represent  a  global  measure  to  be 
satisfied  by  all  types  of  ammunition  and  guns  in  the  system,  ensuring  in  this 
way  that  the  battle  objectives  will  be  met. 

It  should  be  noted  that  many  individual  components  are  either  common 
to  different  guns  or  to  different  ammunitions,  e.g.,  two  different  models  of 
the  same  type  of  gun  may  share  the  same  firing  circuit.  Therefore,  the 
decision  as  to  what  the  minimum  reliability  requirements  are  for  all  the 
components  in  the  system  should  be  taken  globally,  i.e.,  the  result  of  a  model 
that  includes  all  guns  and  all  ammunitions  simultaneously.  In  other  words, 
each  different  round  of  ammunition  and  each  gun  in  the  system  will  have  to 
satisfy  an  equation  of  the  type  E[Tp]  <  1/ef*. 

A  system  like  the  one  described  above  possesses  an  infinite  number  of 
solutions.  It  is  possible,  in  order  to  simplify  the  selection  of  a  particular 
solution,  to  include  an  objective  function,  e.g.,  minimum  procurement  cost. 
This  last  step  is  dependent  upon  knowledge  of  the  different  costs  associated 
with  buying  components  of  different  reliabilities,  information  that  we 
understand  may  not  be  readily  available  to  the  decision  maker. 
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3.  SIMULATION  MODEL 

We  have  developed  a  simulation  model  to  estimate  the  relationship 
between  ef  and  other  standard  performance  measures  which  include  mission 
duration,  average  casualty  rate  over  mission  lifetime,  and  the  percentage  of 
opponent  destructive  power  disabled  at  arbitrary  time  D.  The  model  took  the 
following  aspects  into  account: 

i)  gun  reliability 

ii)  spotting, 

iii)  targets  and  target  hardness, 

iv)  navigation  error, 

v)  guidance  error, 

vi)  fluctuation  in  initial  velocity 

vii)  ships  and  guns  and  dependence,  and 

viii)  all  possible  JMEMS  data. 

The  goal  of  the  simulation  model  was  not  to  simulate  a  particular  known 
engagement,  but  to  give  decision  makers  a  feel  for  the  impact  of  varying  ef 
values  in  terms  they  are  used  to  considering  (standard  performance 
measures).  The  simulation  results  show  that  ef  explains  nearly  all  variability 
in  observations  of  the  percentage  of  targets  destroyed  at  arbitrary  time  D,  the 
time  required  to  destroy  all  the  targets,  and  the  casualty  rate. 

The  simulation  will  also  provide  the  decision  maker  with  the  statistical 
properties  of  these  measures.  Thus  the  decision  maker  has  the  ability  to 
measure  reliability  performance  and  compare  previously  incomparable 
component  configurations. 
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The  decisions  he  will  make  involve  procurement  and  surveillance.  In 
procurement  decisions,  he  will  specify  reliability  values  which  meet 
minimum  requirements  for  ef,  and  which  are  most  economical.  In 
surveillance,  disassembled  rounds  will  be  tested  piece  by  piece.  If  the  round 
still  is  within  the  minimum  ef  standard,  it  is  still  mission  effective.  If  not, 
the  rework  required  to  bring  it  to  mission  effective  status  can  be  prescribed  in 
a  cost-effective  way.   Thus  the  critical  value  of  ef  must  be  established. 

3.1  THE  MODEL 

The  model  is  designed  to  mimic  the  spotting,  communication,  targeting, 
and  destruction  aspects  of  the  NGFS  system.  The  system  consists  of  several 
ships,  each  equipped  with  one  or  more  naval  guns.  These  guns  are  tasked  to 
prosecute  and  destroy  a  prespecified  set  of  targets  by  a  specified  time. 
Interfering  with  the  gunfire  are  inaccuracies  in  the  fall  of  shot,  inaccuracies  in 
the  ship's  internal  navigation  system,  gun  system  interruptions  and  failures, 
and  ammunition  failures.  We  now  describe  the  simulated  processes  in  more 
detail. 
3.1.1     The  Target  List 

The  specified  targets  reside  on  a  target  list  which  gives  each  target's  type, 
the  lethality  points  for  the  targets,  and  each  target's  field  of  view.  The  target 
type  is  referenced  to  a  target  type  list  which  gives  detailed  information  on  the 
method  of  engagement  for  the  target  type.  An  example  of  a  target  type  may  be 
a  TEL  of  an  antiaircraft  gun,  while  a  target  might  be  a  particular  TEL  in  a 
particular  location. 


16 

2/22/91.11:38  AM.16 


Point  values  are  assigned  to  each  of  the  targets  to  control  the  manner  in 
which  they  are  engaged.  Generally  speaking,  targets  are  engaged  in  a  highest- 
points-first  regime,  with  exceptions  made  for  partially  destroyed  targets. 

When  the  NGFS  scenario  begins,  the  targets  with  the  highest  point 
values  are  taken  from  the  target  list  and  engaged,  one  target  per  gun.  Each 
target  is  prosecuted  until  it  is  destroyed,  or  until  the  gun  engaging  it  suffers  a 
severe  casualty.  If  the  target  is  disengaged  and  not  destroyed,  it  is  returned  to 
the  target  list  in  its  current  state  of  damage.  The  scenario  is  not  considered 
complete  until  all  targets  on  the  list  have  been  destroyed. 
3.1.2     Spotting 

Each  target  is  first  engaged  by  using  a  bracketing  method  for  spotting  the 
rounds.  The  brackets  are  established  as  shown  in  Figure  3.1.  Two  points  are 
established  behind  (Pl  for  Plong)  and  in  front  of  (Ps  for  Pshort)  the  target. 
Around  each  point,  a  box  is  established.  The  evolution  starts  as  the  ship 
attempts  to  place  a  round  in  Box  L,  Xl  is  an  acceptable  long  spotting  round. 
Then,  the  ship  attempts  to  place  a  round  in  the  short  box,  say  at  Xs-  The 
distance  from  Pl  or  Ps  to  the  target,  and  the  width  and  length  of  the  spotting 
boxes  are  specified  for  each  target  type.  Note  that  if  the  distance  from  Ps  to  the 
target  is  specified  as  0.0,  the  gun  performs  single-shot  spotting  on  the  target. 

The  preceding  description  fits  the  actions  of  a  ship  engaging  its  first  target. 
If,  however,  the  ship  has  just  destroyed  an  adjacent  target,  this  bracket 
procedure  may  be  overly  cautious  and  time  consuming.  Hence,  if  the  ship 
immediately  engages  a  target  within  the  same  field  of  view  as  the  target  just 
destroyed,  the  ship  performs  single-shot  spotting  instead  of  the  bracketing 
procedure. 
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Figure  3.1.  Target  Spotting 

3.1.3  Firing  for  Effect  (FFE) 

Once  adequate  spotting  has  been  performed,  the  ship  will  fire  for  effect  on 
the  target.  This  involves  firing  at  a  position  exactly  between  the  two  spotting 
locations,  as  at  the  last  single-shot  spotting  location.  The  gun  fires  a  salvo  the 
size  of  which  is  specified  by  the  target  type,  each  round  following  the  previous 
one  by  a  delay  equal  to  the  specified  cycle  time  of  the  gun. 

3.1.4  Destruction 

The  combination  of  the  impact  point,  the  round  type,  and  the  target  type 
allows  us  to  specify  the  amount  of  damage  done  to  the  target.  Each  round- 
target  type  gives  us  an  effective  casualty  radius  (ECR)  within  which  some 
damage  to  the  target  can  be  expected.  Each  target  type  has  a  specified  number 
of  lifepoints,  and  a  number  of  points  it  loses  if  the  round  lands  within  the 
ECR  and  fuses  successfully  called  hitpoints.     We  further  downgrade  the 
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effectiveness  of  the  round  by  a  probability  of  damage,  which  is  conditioned  on 
the  round  landing  within  the  ECR  and  fusing.  Between  the  hitpoints, 
lifepoints,  and  this  conditional  probability,  the  analyst  has  significant 
modeling  flexibility.  When  a  target's  lifepoints  are  no  longer  positive,  it  is 
destroyed. 

3.1.5  Battle  Damage  Assessment 

After  an  effect  salvo  is  fired,  there  is  a  delay  before  the  disposition  of  the 
engaged  target  is  known  to  the  ship.  Contingent  upon  the  condition  of  the 
target,  the  ship  may  do  one  of  three  things.  If  the  target  is  unharmed,  the  ship 
attempts  to  respot  the  target;  if  the  target  is  damaged  but  not  destroyed,  the 
ship  fires  another  salvo  for  effect;  if  the  target  is  destroyed,  the  ship  engages  a 
new  target. 

3.1.6  Navigation  Error  and  Registration  Rounds 

Prior  to  attempting  to  spot  its  first  target,  each  ship  experiences  some  error 
in  its  navigation  solution  as  well  as  some  gun-system  bias.  These  two  effects 
conspire  to  make  the  gun  system  less  accurate.  It  is  a  common  practice  to  fire 
several  registration  rounds  at  a  point  outside  the  target  field,  and  to  track  the 
landing  point  of  these  targets  using  the  ship's  radar.  From  this  data  we  form 
an  estimate  of  the  bias  carried  by  the  ship's  navigation  system  so  that  future 
firings  are  more  accurate. 

3.1.7  Found  Aimpoints  and  Spotting  Inaccuracies 

When  the  ship  fires  a  round,  it  does  so  at  a  specified  aimpoint  (xA,yA) 
given  in  (x,y)  coordinates.  The  round,  due  to  several  exogenous  conditions, 
lands  at  some  nearby  point  (xi,yi).  In  our  model,  we  assume  that  the 
difference  between  (xA,yA)  and  (xi/Yi)  is  a  bivariate  normal  random  variable. 
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With  mean  (0,  0)  and  variance  which  is  specified  in  the  data  set.  Each  gun  is 
allowed  a  different  variance,  and  each  round  type  may  magnify  this 
inaccuracy  as  appropriate. 

When  a  spotter  observes  the  impact  point  relative  to  the  target,  he 
advises  the  ship  to  adjust  the  aimpoint  by  the  observed  error,  as  opposed  to 
the  aimpoint' s  inaccuracy.   See  Figure  3.2. 


Figure  3.2.   Aimpoint  Adjustment 

This  process  continues  until  the  impact  point  satisfies  the  spotting  criteria. 

3.2  GUN  AND  ROUND  FAILURES 

Of  primary  interest  in  this  research  is  the  effect  that  gun  system  and 
ammunition  failures  have  upon  the  overall  performance  of  the  NGFS 
system.  As  discussed  above,  the  system  experiences  failures  which  are  of  two 
distinct  types,  these  being  interruptive  and  noninterruptive  failures.  Each 
failure  is  modeled  to  occur  with  a  given  probability  of  each  firing  of  the  gun, 
and  each  failure  is  assumed  to  be  independent  of  all  previous  firings. 
Furthermore,  the  observation  of  failures  is  hierarchical,  so  that  the 
occurrence  of  a  failure  masks  occurrences  of  failures  of  components  further 
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down  the  detonation  train.     Upon  recovery  from  a  failure,  the  system  is 
assumed  good-as-new. 

3.2.1  Noninterruptive  failures 

Modeling  noninterruptive  failures  is  quite  straightforward.  If  one  of 
these  failures  occurs  during  the  spotting  process,  the  round  is  assumed  to  be 
unobserved  by  the  spotter.  Hence,  the  spotter  will  require  that  an  additional 
spotting  round  be  fired  at  the  current  spotting  aimpoint.  During  the  FFE 
phase  of  the  mission,  the  round  is  simply  ineligible  to  inflict  damage  upon 
the  target. 

3.2.2  Interruptive  failures 

The  duration  of  an  interruptive  failure  changes  depending  on  whether 
the  gun  is  hot  or  not.  Upon  an  interruptive  failure  during  either  phase  of  a 
mission,  there  is  a  possibility  that  the  gun  should  be  taken  off-line  to  be 
repaired,  and  the  engaged  target  should  be  made  available  to  other  guns.  This 
target  release  evolution  occurs  when  the  gun  is  expected  to  be  down  for  more 
than  TOO  LONG  time  units.  The  target  becomes  reengageable  by  another  gun 
after  RELEASE  TIME  units.  The  values  of  TOO  LONG  and  RELEASE  TIME 
are  specified  in  the  scenario. 

If  the  failure  is  not  severe  enough  to  release  the  target,  the  actions  of  the 
system  depend  on  the  mission  phase  at  the  time  of  failure.  If  spotting  is 
interrupted  by  a  failure,  the  spotting  resumes  by  firing  at  the  original  spotting 
location.  If  the  gun  is  firing  for  effect,  the  resumption  of  firing  involves 
shooting  the  entire  salvo  over  again.  However,  when  FFE  is  interrupted  it  is 
possible  that  the  target  has  already  been  destroyed.    Thus,  BDA  is  done  upon 
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every  interruption  of  an  FFE.     An  interruption  of  an  FFE  could,  in  this 
manner,  result  in  reassigning  the  failed  gun  to  a  new  target  upon  recovery. 

3.3  DATA  FILE  PRODUCTION 

The  generation  of  data  for  the  model  falls  into  three  general  areas.  First  is 
the  collection  and  evaluation  of  scenarios.  Second  is  the  translation  of 
known  targets  to  JMEMS  targets  and  the  evaluation  of  munitions  effects. 
Third  is  the  evaluation  of  gun  system  reliability.  All  of  this  data  is  destined 
for  the  scenario  data  file  of  the  simulation  AMMO. 
3.3.1     Collection  and  Evaluation  of  Scenarios 

There  were  three  scenarios  considered  for  use  in  the  study,  based  on 
amphibious  actions  in  the  Middle  East,  northern  Europe,  and  the  northern 
Pacific.  The  Mideast  scenario  is  a  helo-borne  landing  and  therefore  does  not 
meet  the  requirements  of  the  study.  The  Northern  Europe  scenario  is  an 
unopposed  landing  and,  thus,  does  not  meet  the  requirements  of  this  study 
either.  The  third  scenario  was  a  northern  Pacific  scenario  with  many  targets 
designated  to  be  assigned  to  prelaunch  strikes  by  naval  air  and  naval  gun 
assets.  Specifically  18  targets  were  evaluated  as  being  assigned  to  NGFS.  The 
NGFS  time  window  for  these  targets  is  the  48  hours  preceding  the  scheduled 
landing  time.   We  used  this  scenario  as  the  basis  of  our  model  runs. 

Specific  detail  about  the  targets  other  than  location,  elevation  and  general 
nomenclature  was  not  available,  therefore  assumptions  had  to  be  made  about 
the  targets.  It  was  assumed,  for  example,  that  AA  battalion  headquarters 
would  be  fortified  in  such  a  manner  as  to  present  the  damage  profile  of  an 
armored  personnel  carrier.    It  was  assumed  that  each  surface-to-air  missile 
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battery  had  eight  surface-to-air  missiles.    It  was  also  assumed  that  an  aviation 

regiment   would   be   comprised    of   eight   aircraft.      With    regard    to   the 

navigational  error,  it  was  assumed  that  beacons  were  set  in  place  on  the  beach 

for  geographical  reference  and  aid  in  navigation.    Ships  were  assumed  to  be 

able  to  maneuver,  gain,  and  maintain,  a  range  of  12,000  yards  to  each  target  as 

it  was  taken  under  fire. 

3.3.2     Target  Translation 

The  scenario  generated  the  following  target  type  list: 
EW  SITE 
SW  SITE 
GCI  SITE 

AA  BATTALION  HEADQUARTERS 
AA  BATTERY 
HELO  SQUADRON 
SAM  BATTERY 

SAM  BATTALION  HEADQUARTERS 
AVIATION  FIGHTER  SQUADRON 

The  JMEMS  target  list  considered  was  as  follows: 
140MM  ROCKET  AND  LAUNCHER 
FROG  4  ROCKET 
152MM  FIELD  GUN 
ZIL  157  TRUCK 
SEPAL  CRUISE  MISSILE 
CROSS  SLOT  RADAR 

FROG  7B  ROCKET  POINT  AND  AREA  TARGETS 
BMP  ARMORED  INFANTRY  VEHICLE 
PERSONNEL  TARGETS 

For  the  target  translation,  EW,  GCI,  and  SW  sites  were  considered  to  be 

predominantly  antenna  arrays  and  electronics  equipment,  therefore  they 

translated  to  CROSS  SLOT  radar  for  obtaining  input  data  for  the  simulation 


23 
2/22/91.11:38  AM.23 


from  the  JMEMS  publications.  HQs  translated  to  hardened  targets  with  a 
larger  target  radius  than  most  hard  targets.  AA  battery  translated  to  a  152MM 
field  gun.  Helo  and  aviation  squadrons  translated  to  sepal  cruise  missiles. 
SAM  battery  translated  to  FROG  4  rocket. 

From  this  translation  we  were  able  to  generate  the  target  type  list  in  the 
data  file  described  in  Section  4. 
3.3.3     Gun  System  Reliability 

The  5-inch  Gun  Mount  has  53  reliability  blocks  which  are  all  in  series.  A 
reliability  block  diagram  is  a  model  of  a  system  or  operational  mode,  broken 
down  to  a  level  in  which  all  components  in  each  block  are  either  all 
energized  or  all  nonenergized  at  any  point  in  time  (NAVSEAINST  3500. 1A 
Advance  Draft).  Reliability  block  diagrams  (RBD)  are  used  by  the  Fleet 
Analysis  Center  to  determine  reliability  measurements  of  Navy  ship  systems. 
The  RBDs  for  a  particular  system  were  developed  by  the  engineering  and 
maintenance  support  activities  at  the  direction  of  the  cognizant  system 
program  manager. 

The  OPNAV  4798/2K  is  the  primary  source  for  the  Material  Readiness 
Data  Base  (MRDB).  In  addition  the  MRDB  is  augmented  with  casualty  report 
messages,  employment  tapes,  and  other  supplementary  data  sources.  Data 
collected  from  the  start  of  the  data  base  in  January  1984  until  August  1986 
includes  information  from  the  NAVSEA  Non-expendable  Shipboard 
Equipment  Status  Logs,  form  4855/2,  which  are  no  longer  required.  Data 
from  1984  to  1986  was  used  because  it  includes  the  equipment  status  log 
information.  It  is  a  source  of  continuous  real  time  observation  and  actual 
clock  readings.    The  mean  time  between  failures  (MTBF)  and  the  mean  time 
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to  repair  (MTTR)  were  obtained  for  each  of  the  53  RBD's  from  the  MRDB. 
The  RBD  MTBFs  and  MTTRs  were  used  instead  of  the  system  reliability  to 
avoid  the  data  base  scenario  adjustments  of  system  reliability  that  are  made 
using  a  demand  factor  for  each  of  the  RBDs  based  on  the  percentage  of  time  it 
is  energized  during  a  typical  mission  life  normalized  by  a  ratio  of  energized 
time  to  real  clock  time,  aggregated  over  all  units  reporting.  We  wanted  to  use 
as  pure  a  number  as  possible  and  therefore  modeled  the  individual  53  RBDs 
in  the  simulation  and  used  the  unadjusted  data  from  the  1984  to  1986  time 
frame. 
3.3.4     Data  Collection  Postmortem 

There  is  a  mismatch  between  those  targets  that  we  are  describing  in  our 
scenarios  and  the  information  about  the  specific  effect  of  our  ammunition  on 
targets  that  we  were  able  to  develop  at  this  time.  We  are  in  the  process  of 
getting  the  effects  data  needed  from  the  JMEM  Munition  Effects  Working 
Group.  Our  scenarios  should  be  generated  with  the  same  level  of  granularity 
or  scale  as  the  JMEMS  publications,  so  that  translation,  if  required,  can  be 
done  in  reasonable  fashion.  In  terms  of  our  own  systems,  there  needs  to  be  a 
methodical  way  to  collect  information  in  order  to  support  reliability 
calculations,  and  to  do  so  in  such  a  way  as  to  isolate  those  periods  of  time 
when  the  systems  are  specifically  in  a  mission  state.  It  does  no  good  for  us  to 
evaluate  our  systems  at  90  percent  reliability,  lets  say,  according  to  the  data 
base  when  the  system  does  not  work  50  percent  of  the  time  while  it  is  on  the 
firing  line.  System  reliability  for  a  mission  may  not  be  accurate  for  the  mark 
45  mod  1  system  because  the  collected  data  is  based  on  clock  time,  not  mission 
time. 
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Because  of  these  factors,  it  was  deemed  valuable  that  NGFS  be  observed  so 
that  some  idea  of  validation  of  the  model  could  be  obtained.  The  observation 
of  one  ship  does  not  suffice  to  validate  the  simulation  model,  but  it  did  not 
invalidate  the  model.  It  was  encouraging  to  see  that  the  rate  of  fire  of  the  ship 
in  actual  clock  time  was  similar  to  that  predicted  by  the  model.  It  was  also 
encouraging  to  see  the  gun  perform  as  well  as  it  did  as  the  ship's  crew  became 
acclimated  to  the  environment  of  NGFS. 

4.0  RUNNING  THE  SIMULATION 

In  this  section,  we  describe  the  data  files  required  to  operate  the 
simulation  model,  and  the  output  attainable. 

4.1  DATA  FILES 

There  are  two  data  files  required  to  run  the  simulation  AMMO.  These 
files  are  known  as  the  scenario  and  run  control  files.  The  scenario  file,  an 
example  of  which  is  shown  in  Figure  4.1,  contains  three  distinct  sections: 
timing  information,  target  description  information,  and  ship-gun-round 
system  description  information.  Finally,  there  is  a  list  of  seeds  used  in  the 
simulation. 

All  of  the  data  collected  from  the  FLTTAC,  JMEMS,  and  other  sources  is 
destined  for  the  scenario  data  file.  The  scenario  supplied  by  NWSC  was  used 
to  choose  the  targets  and  target  types,  and  to  specify  the  numbers  of  ships  and 
guns.  In  addition,  the  data  on  the  timing  and  accuracy  of  the  gun  system  was 
provided  by  rough  estimates  from  data  collected  aboard  the  USS  Ainsworth. 
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TIMING  INFORMATION 

5.5      bdatim,  the  time  (in  min.)  it  takes  to  do  BDA 
3.5      spttim,  the  time  between  spotting  round 
2         numgun,  the  number  of  guns  in  the  scenario 

1  numshp,  the  number  of  ships  in  the  scenario 

2  numshl,  the  number  of  shell  types  used  in  the  scenario 
4         numtgt,  the  number  of  targets 

2         numtty,  the  number  of  target  types 
100.    dtime,  the  time  of  the  amphibious  assault 

1         dswtch,  1  means  the  simulation  stops  at  dtime,  0  means  destroy  all  targets 
4         regmd,  number  of  registration  rounds  each  ship  shoots  at  the  beginning  of  each 
scenario 
TARGETS — id     tgttyp  points  fov  (one  for  each  target) 
.one     1  .  100         1 

two     1         "99  2 

three  2  45  1 

four    2  56  2 

TARGET  TYPES — id,  shltyp,  lifpts,  hitpts,  effect,  ecr,  deflct,  jmemsp,  ecrlng,  pcorct,  blng, 
bwd 

type  11  10         3  5  50        20         0.90      1  1.0       1.5        1.0 

type  2  1         10         3  5  50        20         0.90      1  1.0       1.5        1.0 

SHIPS — ship  nav  error  sigshp  (.,  1)  sigshpC,  2) 
100.0    100.0  standard  errors,  x  and  y,  of  the  ship  nav  system 

GUNS — ship,  gnumf,  hotrnd,  siggun(.,  1),  siggunC,  2),  cycle  (one  per  gun) 
gfailp,  gfailt(cold,  hot)  (one  per  failure  mode  of  the  gun) 
1         2  10         20.0      60.0      0.05 

0.05    1.0        2.0 
0.10    0.0       0.0 

1         2  10         20.0     60.0      0.05 

0.05    1.0       2.0 
0.10    0.0       0.0 
SHELLS — snumf,  sigshlC,  1),  sigshl(.,  2)  (one  per  shell  type) 
sfailp,  sfailt(hot  and  cold)  (one  per  shell  failure  mode) 
4  1.0        20.0 

0.5       5.0        10.0 
0.01      0.0        0.0 
0.03      3.0        3.0 
0.002    4.0        14.0 
3  1.0        0.0 

0.01  0.0  0.0 
0.03  3.0  3.0 
0.002   4.0        14.0 


Figure  4.1.  Scenario  Data 
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The  blocks  of  data  pertaining  to  the  guns  and  shells  may  be  nonintuitive 
in  their  design.  The  first  line  for  each  gun  description  gives,  in  order,  the 
ship  on  which  the  gun  is  mounted,  the  number  of  failure  types  for  the  gun, 
the  number  of  rounds  fired  before  the  gun  is  considered  hot,  the  standard 
errors  of  the  dispersion  of  shot  for  the  gun  (x  and  y),  and  the  time  between 
rounds  fired  for  effect  for  the  gun.  The  next  several  lines  of  data  are  the 
probability  of  occurrence  and  the  recovery  times  (cold  and  hot)  for  each 
failure  type. 

The  blocks  of  data  pertaining  to  the  shell  types  are  structured  similarly. 
The  first  line  gives  the  number  of  failures  for  the  shell  type,  followed  by  the 
additional  dispersion  caused  by  the  round.  The  following  lines  give  the 
probability  of  occurrence,  and  the  recovery  time  (cold  and  hot)  for  each  failure 
type. 

A  second  file,  the  run  control  data  file,  contains  some  simple  information 
used  to  limit  the  runs  of  the  simulation  and  to  implement  our  experimental 
design.  An  example  of  the  run  control  file  is  given  in  Figure  4.2.  The  run 
control  file  specifies  the  maximum  duration  of  any  single  simulation  run, 
MAXTIME,  the  level  of  accuracy  required  of  each  of  the  performance 
measures,  PCT,  and  the  maximum  number  of  replications  a  given 
experiment  is  allowed  to  run,  MAXREPS.    For  example,  if  MAXREPS  is  200 

and  PCT  is  0.05,  then  the  simulation  experiment  will  stop  when 

1)  the  confidence  interval  half  width  is  less  than  5%  of  the  mean  for  each 
performance  measure,  or 

2)  200  replications  have  been  completed. 

Lines  4-6  of  the  file  concern  the  "scrunching"  of  ECR  values.  As  has  been 
mentioned  in  the  discussion  of  JMEMS  data,  it  was  often  seen  that  a  given 
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round-target  pair  would  have  an  effective  casualty  radius  of  850  meters  and  a 
probability  of  destruction  of  0.01.  We  found  this  unreasonable  and  devised  a 
crude  way  to  make  the  probability  of  destruction  increase  by  insisting  on  more 
accuracy  in  round  delivery. 


200 

MAXREP 

35,000.00 

T 

0.15 

PCT 

F 

T  TO  MODIFY  THE  DISKS,  F  TO  SKIP  THAT  PORTION 

40.0 

R  (=  ADAPTIVE  RADIUS  WE  USE  TO  SCRUNCH  ECRS) 

1         0.000 

0.0005 

1         0.00 

0.004 

1         0.00 

0.01 

1         0.00 

0.004 

1         0.000 

0.002 

Figure  4.2.  Run  Control  File 

We  made  the  assertion  that,  within  limits,  ECR  x  Pr[destruct]  =  C  for 
some  constant  C  corresponding  to  the  firepower  of  the  round  against  the 
target.  The  aforementioned  limits  are  established  in  lines  4  and  5  of  the  data. 
R  gives  the  minimum  ECR  value  for  any  round-target  combination,  while 
PMAX  gives  the  maximum  Pr[destruction].  The  program,  when  instructed, 
will  attempt  to  concentrate  firepower  until  one  of  these  bounds  becomes 
tight.  Experimentally,  we  have  found  that  the  simulation  predicted  highest 
gun  efficiency  when  R  was  equal  to  the  round  dispersion  standard  error  given 
in  the  scenario  data  file. 

The  last  several  lines  give  the  experimental  design,  which  is  of  the 
complete  factorial  type.  We  are  interested  in  the  behavior  of  each  of  our 
performance  measures  when  the  probabilities  of  failure  of  each  of  the  round 
components  are  varied.    By  convention,  we  vary  the  probability  of  failure  for 
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round  type  1  in  the  experiment.  If  the  scenario  involves  several  round  types, 
the  user  should  manipulate  the  scenario  data  file  so  that  round  type  1  is  the 
round  of  primary  interest  for  the  run. 

The  design  is  given  by  the  number  of  levels,  the  lower  limit,  and  the 
upper  limit  of  the  probability  of  failure  for  each  failure  type  for  round  type  1. 
Thus,  if  the  line  corresponding  to  failure  type  1  in  the  run  control  file  is  given 
as 

3  0.01      0.03, 

then  type  1  failure  probabilities  will  vary  over  the  set  {0.01,  0.02,  0.03}.    If  the 
number  of  levels  for  a  failure  type  is  given  as  1,  the  probability  of  failure 
reverts  to  that  found  in  the  scenario  data  file. 
4.2        Model  Responses 

When  the  model  runs,  the  user's  screen  is  notified  of  the  number  of 
experiments  that  will  be  performed  in  the  run,  the  number  completed  thus 
far,  and  the  number  of  replications  finished  for  the  current  experiment. 
Upon  completion  of  each  experiment,  the  user  is  notified  of  the  number  of 
replications  of  the  experiment  which  were  censored  because  the  targets  were 
not  all  destroyed  before  MAXTIME.  The  number  of  censored  runs  should  be 
small  (ideally  0!)  for  each  experiment.  If  this  number  becomes  a  significant 
portion  of  the  sample  size  for  the  experiment,  the  values  of  the  scenario, 
failure,  or  MAXTIME,  should  be  reconsidered. 

When  the  simulation  does  run  successfully,  the  output  is  given  in  the 
form  shown  in  Figure  4.3.  Estimates  of  the  primary  measures  of 
performance,  the  mean  scenario  completion  time,  the  time  average  of  the 
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KEEPON  =  F 


0  OF 


10REPS  CENSORED 


THE  NUMBER  OF  REPLICATIONS  REQUIRED  WAS 

THE  AVERAGE  MISSION  LIFETIME  = 

VARIANCE  OF  THE  MEAN  = 

STANDARD  DEVIATION  OF  THE  MEAN  = 

THE  TIME  AVG  OF  THE  TARGET  VALUE  = 

VARIANCE  OF  THE  MEAN  = 

STANDARD  DEVIATION  OF  THE  MEAN  = 

ROUNDS  PER  MINUTE  BY  GUN 


1 

1 

289 

WITH 

STD 

ERROR 

.146 

2 

1 

612 

WITH 

STD 

ERROR 

.101 

3 

1 

326 

WITH 

STD 

ERROR 

.148 

4 

1 

387 

WITH 

STD 

ERROR 

.124 

5 

1 

338 

WITH 

STD 

ERROR 

.120 

6 

1 

722 

WITH 

STD 

ERROR 

.149 

THE  GLOBAL  AVERAGE 
1.446 

THE  AVG 


WITH  STD  ERROR 


059 


OF  TARGETS  DEAD  AT  DTIME  = 
VARIANCE  OF  THE  MEAN  = 
STANDARD  DEVIATION  OF  THE  MEAN  = 

THE  AVG  SURVIVING  TGT  VAL  AT  DTIME  = 

VARIANCE  OF  THE  MEAN  = 

STANDARD  DEVIATION  OF  THE  MEAN  = 

GUN     1  THE  AVG  ROUNDS  PER  MISSION  = 

VARIANCE  OF  THE  MEAN  = 
STANDARD  DEVIATION  OF  THE  MEAN  = 

GUN     2  THE  AVG  ROUNDS  PER  MISSION  = 

VARIANCE  OF  THE  MEAN  = 
STANDARD  DEVIATION  OF  THE  MEAN  = 

GUN     3  THE  AVG  ROUNDS  PER  MISSION  = 

VARIANCE  OF  THE  MEAN  = 
STANDARD  DEVIATION  OF  THE  MEAN  = 

GUN     4  THE  AVG  ROUNDS  PER  MISSION  = 

VARIANCE  OF  THE  MEAN  = 
STANDARD  DEVIATION  OF  THE  MEAN  = 

GUN     5  THE  AVG  ROUNDS  PER  MISSION  = 

VARIANCE  OF  THE  MEAN  = 
STANDARD  DEVIATION  OF  THE  MEAN  = 

GUN     6  THE  AVG  ROUNDS  PER  MISSION  = 

VARIANCE  OF  THE  MEAN  = 
STANDARD  DEVIATION  OF  THE  MEAN  = 
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563.5317 

1146.4990 
33.8600 

77.2558 

23.3675 

4.8340 


1.0000 
.0000 
.0000 

.0000 
.0000 

.0000 


456.8999 


2833 . 6540 
53.2321 


473.2000 


3422.3510 
58.5009 


531.3000 


3943.9120 
62.8006 


440.3000 


3836.4900 
61.9394 


402.5000 


4145.2280 
64.3834 


487.2000 


5527.7960 
74.3491 


Figure  4.3.  Example  Output  File 
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surviving  target  value,  and  the  target  value  surviving  at  DTIME  are 
prominently  displayed.  In  addition,  we  report  several  sets  of  statistics 
concerning  the  experiences  of  each  gun  used.  This  report  is  generated  by  the 
program  only  when  the  number  of  experiments  in  the  design  equals  1.  That 
is,  only  when  a  simple  simulation  run  using  the  scenario  data  values  is  done. 
When  there  is  more  than  one  experiment  in  a  design,  the  results  are  given  in 
data  files  amenable  to  input  in  a  graphical  statistical  analysis  package  such  as 
GRAFSTAT. 
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5.0  DETERMINING  RELIABILITY  GOALS 

In  the  previous  sections,  we  have  produced  descriptions  of  the  analytic 
reliability  measure  ef,  and  of  the  simulation  analysis  system.  In  the  most 
fundamental  terms,  ef  is  a  simple  function  of  reliability  measures  of 
individual  ammunition  and  gun  system  components.  The  simulation  also 
uses  the  reliability  data,  as  well  as  other  system  performance  characteristics 
and  an  NGFS  scenario  as  input.  The  output  of  the  simulation  is  a  set  of  battle 
goal  results,  as  listed  in  Figure  4.3.  Let  us  denote  these  k  battle  goal  values  by 
the  vector  bg(bgi,  bg2, ...,  bgk). 

The  reason  for  developing  the  simulation  model  is  that  components  of  bg 
have  relevance  and  meaning  to  our  prospective  decision  maker.  Given  a 
vector  of  battle  goal  results,  we  expect  the  decision  maker  to  be  able  to 
determine  if  the  performance  of  the  system  is  acceptable.  Pursuing  this 
expectation,  we  compel  the  decision  maker  to  specify  a  function 

¥:  9tk  ->  {0,1} 

so  that  ^(bg)  =  1  if  and  only  if  the  outcome  bg  is  acceptable  in  terms  of 
performance,  ^(bg)  =  0  if  not.  For  simplicity,  we  will  assume  that  W  has  the 
following  form: 

jl       iff  ha  <  be*  i  =  l,  2, ...,  k 
¥(bg)=  &       B  (5.1) 

(0       otherwise. 

(     *  *\ 

Hence  bg*  =   bgl7  bg^, ...,  bgk    is    the  decision  maker's  threshold  performance 

vector.      Obviously   some   sign-switching  will  be  necessary   so   that  this 

threshold  vector  makes  physical  sense. 
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We  propose  that  bg  is  a  function, 

bg:[0,l]->$Rk 

so  that  each  battle  goal  is  a  function  of  ef,  bg(e/)  e  SRk.    Hence,  we  submit  the 
following   procedure   for   establishing   reliability   goals   for   major   caliber 

ammunition  components: 

1.  specify  a  set  of  N   experimental   reliabilities   for   each   ammunition 
component  for  each  round  type. 

2.  specify  bg*; 

3.  i  =  l,2,  ...,N 

i.      Calculate  efc 

ii.     estimate  bg(e/i)  via  AMMO; 

4.  ef*  =  mm{efi:Y(b&(efi))  =  l} 

i     L  J 

The  validity  of  the  above  procedure  is  based  on  the  assumption  that  bgi  is 

nondecreasing  in  ef  for  each  i. 

Example  2. 

From  our  preliminary  analysis,  we  chose  the  battle  goals  as  follows: 

bgi   =  total  mission  time; 

bg2  =  total  firepower  of  threat  at  1  hour  after  engagement  begins 

bg3  =  time-integrated  point  value  of  targets 

bg4  =  percentage  of  targets  left  undestroyed  at  time  of  invasion,  48  hours 
after  engagement  begins. 

Obviously  the  best  of  all  possible  situations  is  to  have  bg  =  [0,0,0,0]T. 

We  expect  the  decision  maker  to  specify  the  threshold  vector  bg*  so  that 

we  can,  in  turn,  provide  a  threshold  ef*.     Obviously,  lower  values  of  the 

components  of  bg*  will  face  the  value  of  ef*  up.   The  usefulness  of  ef*  is  that 

it  relates  directly  to  gun  and  ammunition  component  failure  probabilities. 
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5.1   ENHANCING  EFFECTIVENESS 

Given  that  the  decision  maker  has  specified  a  threshold  level  of 
performance  bg*,  how  must  we  deal  with  this  threshold?  For  each  round 
type,  we  are  required  to  meet  the  attendant  effectiveness  threshold  ef*.  The 
round  types  are  considered  to  operate  in  a  fixed  environment,  including  the 
reliability  performance  of  the  gun  system  the  round  is  used  with.  Thus,  of  all 
of  the  system  component  reliabilities  that  impact  the  calculation  of  ef  for  a 
round  type,  only  the  round  component  reliabilities  are  under  our  control. 

Let  us  consider  one  round  type  with  round  components  1,  2,  ...,  k  and 
exogenous  components  k  +  1,  k  +  2,  ...,  N  the  exogenous  components  being 
those  of  the  intended  gun  system.  We  hope  to  provide  round  to  the  fleet 
such  that  pi,  p2,  ...,  pk-  The  k  probabilities  of  failure  for  each  of  the  round 
components  are  small  enough  so  that  ef*  >  ef*.  If  this  is  not  the  case,  then 
one  or  more  of  the  component  reliabilities  must  be  improved  in  order  for  the 
round  to  conform  to  the  specified  reliability  threshold. 

The  decision  maker  we  allude  to  actually  performs  two  roles,  he  procures 
ammunition  components  for  assembly  with  currently  held  components  to 
make  new  rounds,  and  he  monitors  the  reliability  of  existing  stockpiles  of 
ammunition.  When  monitoring  the  stockpiles,  he  decides  whether  or  not  to 
perform  some  set  of  rework  upon  an  ammunition  lot  to  improve  the 
reliability  performance  of  the  lot. 
5.1.1     The  Procurement  Decision 

For  the  procurement  process,  we  will  assume  that  the  decision  maker 
currently  has   in  place  procurement   mechanisms   for  k-1   of  the  round 
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components,  and  has  a  high  degree  of  confidence  in  the  value  of  Xj, 
j  =  1,  2,  ...,  N,  j  *  i,  the  failure  rates  of  the  individual  components.  He  needs  a 
reliability  goal  for  component  i,  that  is,  he  needs  to  specify  the  maximum 
allowable  Xj  which  will  produce  a  round  conforming  to  the  overall  reliability 
threshold. 

From  Section  2.3,  we  can  state  the  reliability  threshold  constraint  as 


ef*< 


N 


-i-l 


lTy(l-py)/p/+1 


(5.2) 


Isolating  pi,  this  constraint  becomes 


A^lSP 


(5.3) 


where  A  is  given  by 


A  = 


\ 

H 


ifczal+i 


\}*' 


Tf\ 


(5.4) 


Tracing  pj  to  its  origins  as  a  function  of  the  failure  rate  Xj,  we  arrive  at  the 
constraint 


(1-1/04-1)) 
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(5.5) 


This  is  our  reliability  goal  for  component  i. 

We  have  related  the  constraint  of  the  effectiveness  of  a  round  to  the 
failure  rate  of  the  component  to  be  procured.   We  recommend  that  acceptance 
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tests  for  the  new  component  establish  x}  as  the  upper  control  limit  for  the 
percent  defective  in  all  procured  lots  of  component  i. 

Finally,  if  component  i  is  to  be  used  to  build  rounds  of  several  different 
types,  the  reliability  goal  should  be  set  by  the  round  type  establishing  the 
lowest  Xj. 
5.1.2     The  Surveillance  Decision 

Ammunition  surveillance  is  the  practice  of  removing  several  rounds 
from  a  stockpiled  lot  of  ammunition  and  testing  the  rounds.  This  testing  is 
done  in  two  modes,  these  being  total-system  and  component-wise.  Currently, 
the  acceptance  criteria  for  pronouncing  a  stockpiled  lot  fleet-ready  are  ad  hoc. 
We  propose  replacing  the  current  surveillance  test  criteria  with  those  that 
will  ensure  that  the  lot  tested  exceeds  ef*  in  efficiency. 

Usually,  only  a  subset  of  the  components  in  a  round  are  selected  for 
component-wise  surveillance  testing.  From  the  tests,  we  can  easily  establish 
an  upper  confidence  limit  on  the  value  of  Xj  for  each  round  component  i 
which  is  tested. 

Components  that  cannot  or  are  not  tested  component-wise  are  tested  in 
the  total-system  test.  During  the  total  system  test,  the  number  of  failures 
which  is  interruptive  should  be  counted  and  classified,  so  that  upper  control 
limits  are  established  for  p;,  the  expected  number  of  failures  of  type  i  which 
occur  between  successful  rounds.  Failures  of  types  which  are  attributed  to 
components  tested  component-wise  are  used  to  sharpen  the  estimates  of  the 
failure  rates  for  these  components. 

Thus,  after  the  initial  surveillance  tests,  we  will  have  an  estimate  of  ef  for 
the  lot.     If  this  estimate  is  above  ef*,  we  allow  the  lot  to  remain  in  the 
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stockpile  and  proclaim  the  lot  to  be  conforming  to  our  reliability  goal.  If  the 
lot  is  not  conforming,  the  decision  maker  has  the  option  of  replacing  one  or 
more  of  the  components  in  each  round  in  the  lot.  He  has  knowledge  of  the 
failure  rate  of  the  replacement  components  he  has  at  hand,  so  he  may 
determine  whether  performing  the  replacement  will  bring  the  lot  into 
conformance  with  the  reliability  goal.  No  doubt,  this  rework  decision  has  an 
economic  aspect  we  do  not  address. 
5.1.3     Evaluating  Marginal  Improvements 

As  we  have  developed  ef  as  our  performance  measure,  it  is  incumbent 
upon  us  to  exploit  the  functional  nature  of  ef  to  determine  relationships 
between  incremental  improvements  in  reliability,  training,  or  operations  and 
fleet  effectiveness  in  NGFS.   We  do  this  by  taking  partial  derivatives. 

Consider  the  following  two  equations; 

def    Vi -I    „ 

wrryr'f2  (5-6) 

def    Ti 


^p,     Pi 


ef1  (5.7) 


As  0  <  pi,  ef  <  1,  we  know  that  def/dT-,  e  [-1,0],  while  def/dp^  e  [0,1]. 

Equation  1  primarily  addresses  training  of  crew  members,  as  better  trained 
crews  will  shorten  recovery  times  from  gun  or  ammunition  failures. 
Equation  1  tells  us  that  the  greatest  operational  payoff  comes  from  reducing 
the  recovery  time  of  failures  which  occur  most  frequently,  that  is,  with  low 
values  of  pj.  Note  further  that  Equation  1  does  not  explicitly  involve  Tj,  so 
that  we  see  clearly  that  frequency  of  failure  is  more  important  than  severity. 


38 
2/22/91.11:38  AM.38 


Equation  2  tells  us  that  incremental  improvements  in  reliability  of 
individual  components  are  ordered  by  the  ratio  Tj/pj.  The  greater  this  ratio, 
the  more  important  an  incremental  improvement  in  Xj  will  be.  Note  that 
Tj/pj  is  large  when  Tj  is  large  and  . . . 

Xj  is  large.  The  upshot  of  this  discussion  is  that 

i)  crews  should  train  with  the  goal  of  reducing  the  recovery  time  for  the 
most  frequent  failures; 

ii)  contractors,  and  procurement  agents  should  weigh  incremental 
improvements  in  component  reliability  using  the  ratio  Tj/pi. 

Conclusion  (ii)  has  direct  impact  on  determining  bonus  values  for  contractors 

who  exceed  reliability  specifications  in  their  contracts. 

5.2        Output  Analysis 

An  experiment  was   designed  whose  purpose  was   to  determine  the 

functional  relationship  between  each  of  the  battle  goals  and  the  value  of  ef. 

The  experiment  consisted  of  three  steps: 

i)  the  simulation  for  K  of  different  combinations  of  round  component 
failure  rates,  executing  M  replications  per  combination; 

ii)  for  each  of  the  KM  runs,  exploit  the  difference  between  the  observed 
value  of  ef  and  its  known  expectation  to  adjust  the  battle  goal  outcome 
for  the  run; 

iii)  develop  a  set  of  regression  metamodels  used  to  determine  the 
functional  relationship  between  each  battle  goal's  expected  value  and 
the  theoretical  value  of  ef. 

The  experiment  was  performed  for  K  =  34  and  M  =  200,  using  a  stylization 

of  a  scenario  provided  by  NWSC.   The  data  used  was  perturbed  from  realistic 

values   so   as   to   avoid   any  classification   constraints.      In  the  following 

discussion,  we  will  treat  the  following  two  battle  goals, 
i)      bgi  =  mean  mission  duration; 
ii)    bg2  =  time  average  of  enemy's  firepower; 
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where  firepower  is  given  by  each  target's  value,  see  Section  3.1.1.  A  direct 
relationship  between  the  time  average  of  the  enemy's  firepower  and  expected 
troop  loss  can  be  made  if  one  (unrealistically)  expects  the  amphibious  assault 
to  begin  at  the  beginning  of  the  scenario. 

Figures  5.1  and  5.2  show  the  outcome  of  our  experiment.  In  each  figure, 
each  datum  represents  the  average  response  from  the  200  runs  of  one  failure 
rate  combination,  after  adjustment  as  described  in  Step  ii.  This  adjustment 
was  performed  using  the  linear  control  variate  method,  see  Bratly,  Fox,  and 
Schrage  [1983]. 

Briefly,  this  method  consists  of  modifying  each  response  from  the 
simulation  as 

%  =  Vi-PH-ef\  (5.8) 

where  Yj  is  the  ith  replication  of  the  random  variable  Y,  and  efc  is  the  ith 
observation  of  the  random  variable  EF.  Since  E[EF-ef]  =  0,  we  have 
E[  Y]  =  E[Yj]  for  each  i.  it  is  well  known  that  the  best  choice  of  [5  is  given  by 

cov[£F,Y]^EgL  (5.9) 

var[£FJ 

as  we  don't  know  the  value  of  p,  we  estimate  the  required  quantities  from 
each  of  the  K  experimental  combinations  of  failure  rates.  These  estimates  are 
shown  graphically  in  Figure  5.3.  Figure  5.4  graphically  illustrates  the 
reduction  in  variance  in  the  mean  response  of  each  of  the  experimental 
combinations  for  the  time-averaged  fire  power  battle  goal.  The  reductions  in 
variance  are  in  the  neighborhood  of  40-50%  for  this  battle  goal.  Thus,  by 
exploiting  the  difference  between  the  simulation's  experimental  outcome  for 
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ef  and  the  true  expected  value,  we  have  made  our  procedure  40-50%  more 
efficient. 
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Figure  5.1.  Battle  Goal  1  Plotted  as  a  Function  of  Theoretical  ef 

For  the  two  battle  goals,  we  see  the  fit  of  the  two  regression  metamodels 

we  constructed  using  standard  techniques,  see  Kleijnen  [1975].    Our  estimates 

of  the  linear  relationship  between  ef  and  each  of  the  battle  goals  is  given  as 
bgi  =  427.62  -  415.79  ef  (5.10) 

bg>  =  1693.2  -  1069.4  ef  (5.11) 

Both  fits  were  subjected  to  standard  regression  diagnostics,  and  both  were 

determined  to  be  acceptable. 
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Figure  5.2.  Battle  Goal  2  Plotted  as  a  Function  of  Theoretical  ef 

It  is  worth  noting  that  with  our  combination  of  variance  reduction  technique 
and  linear  metamodel,  we  are  regressing  our  battle  goal  outcomes  on  the 
quantities  efi  -  ef  and  ef,  and  that  these  two  calculations  are  pursued  for  very 
different  ends. 
Setting  Reliability  Goals,  an  Example 

Suppose  that  the  decision  maker  makes  the  determination  that  for  the 
targets  and  scenario  described,  that  the  battle  goal  threshold  is  bg*  =  (100,860). 
That  is,  the  mission  execution  is  determined  to  be  below  acceptable  levels  of 
performance  if  it  takes  100  minutes,  or  if  the  time-integrated  fire  power  score 
is  greater  than  860.  Using  the  regression  metamodels,  we  see  that  executing 
the  mission  time  constraint  implies  that  ef  >  0.788,  while  meeting  the  fire 
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power  constraint  implies  that  ef  >  0.779.    Thus,  from  this  simple  analysis,  we 
can  conclude  that  the  threshold  ef  is  given  as  0.788. 
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6.0  CONCLUSION 

What  we  have  endeavored  to  provide  to  the  managers  of  the  U.S.  Navy's 

major  caliber  ammunition  program  are  the  following: 

i)  a  measure  of  performance  which  relates  defect  rates  of  ammunition 
components  to  battle  force  performance; 

ii)  a  simulation  model  to  assist  government  analysts  in  establishing 
threshold  values  for  the  performance  measure; 

iii)  a  methodology  for  prescribing  maximum  defect  rates  for  ammunition 
components  to  ensure  that  new  and  stockpiled  ammunition  conforms 
to  specified  threshold  levels. 

Hence,  we  have  an  important  set  of  tools  useful  in  managing  procurement 

and  surveillance  of  ammunition. 

It  is  our  view  that  these  tools  should  be  integrated  into  a  system  of 

evaluation,  physical  experiment,  data  production,  and  procurement  and 

surveillance  decisions.    The  weakness  of  the  tools  we  have  produced  thus  far 

are  that  the  simulation  model  is  not  verifiable — we  cannot  match  simulation 

output  with  that  of  a  physical  experiment.   Figure  6.1  shows  our  concept  of  an 

integrated  reliability  performance  evaluation  system.     What  follows  are 

remarks  concerning  each  component  of  this  system. 

6.1  LIVE  FIRE  EXPERIMENTS 

Naval  forces  use  their  major  caliber  guns  in  exercises  and  qualification 
tests  and  equipment  acceptance  tests  throughout  the  year.  The  gun 
component  reliability  and  recovery  time  data  available  through  current 
sources  are  not  very  complete  and  do  not  address  the  needs  of  performance 
analysis  of  ammunition.     The  establishment  of  our  analysis  tools  should 
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compel  the  major  caliber  ammunition  program  to  identify  and  collect  data 
from  these  live  fire  experiments,  especially  with  regard  to  gun  system  failure 
rates  and  recovery  time. 


Assessment 
Process 


A.  Live  Fire 
Experiment 


D.  Procurement/ 
Surveillance 
Decisions 


B.  Produce 
Scenario 
Data  from 
Outcome 


C.  Simulate  to 
Produce 
Operational 
Evaluation 


Figure  6.1.  Integrated  Reliability  Performance  Evaluation  System 

6.2  PRODUCE  SCENARIO  DATA  FROM  OUTCOME 

Using  the  data  from  the  live  fire  experiments,  scenarios  data  files  should 
be  produced.  See  Section  3.1. 

6.3  SIMULATE  TO  PRODUCE  OPERATIONAL  EVALUATION 

The  produced  data  files  should  be  subjected  to  the  same  analysis  process 
we  have  executed  in  Section  4  for  the  NSWC-supplied  scenario.  From  this 
analysis,  reliability  thresholds  are  established  for  use  in  the  management  of 
procurement  and  surveillance  processes. 
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6.4  PROCUREMENT/SURVEILLANCE  DECISIONS 

Using  the  threshold  values  established,  the  management  decisions 
required  are  made.  This  will  result  in  ammunition  getting  to  fleet  user 
which  has  been  affected  by  this  process.  The  ammunition  will  thus  be  used  in 
subsequent  live  fire  experiments.  Hence,  there  will  exist  a  feedback  from 
these  decisions  to  the  evaluation  of  ammunition.  The  process  will  become 
self-tuning. 
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